I joined Pinecone as a Solutions Engineer about a year ago. Since then, several of my projects have turned into internal tools and examples for testing different parts of the platform. This article is the first in a series walking through them; later posts will cover my Pinecone Index Migrator, a GPU benchmarking utility, a dense vector embedding generator, and examples for querying dense, sparse, and hybrid indexes.It starts with something that sounds simple but matters more at scale: generating test data. For most of my work I need a repeatable way to create datasets that are large enough to be useful, realistic enough to support recall and accuracy testing, and flexible enough to reuse across experiments. That led to a small collection of utilities for generating Parquet files, creating embeddings, preparing metadata, and importing the data into Pinecone.The utilities are broken into a few parts:Parquet file generatorVector embedding generatorParquet categorizer using a local LLMParquet repartitioner that uses categories as namespaces for Pinecone bulk importThis article focuses on the first part of that workflow: generating the source data and preparing it for embedding and import. The full, runnable scripts live in the companion notebook; the snippets below are the parts worth discussing in line.Why Test Data Generation MattersOne of the first things I ran into when I started working with Pinecone was the need to create different datasets for different types of tests. Not all test data is the same, and not all dataset sizes are the same.It is easy to say that a platform works "at scale," but that phrase means different things depending on the workload. A test with 500,000 vectors isn't the same as one with 15 million — or 100 million, 500 million, or a billion. Each size introduces its own set of problems around ingestion, storage, latency, filtering, cost, and operational workflow.There are some great sources of raw data out there, including Hugging Face datasets and Kaggle datasets. Those are a good starting point, but they are only part of the problem. For vector search testing, I still need to generate vectors. That leads to an important question:Do the vectors need to be real?Sometimes no. If I am only testing write throughput, storage behavior, or query latency, random vectors are good enough — I've had use cases where speed was the only thing under evaluation and the meaning of the vector did not matter. But for most tests, recall and accuracy do matter, and that means the embeddings have to represent the actual text. Random vectors will not help there.So for this project, I wanted to generate a realistic dataset from real text, create real embeddings, and store everything in a format that could be imported into Pinecone efficiently.Requirements for the DatasetFor this first utility, I wanted a reference dataset that I could regenerate as needed. I also wanted the data to be familiar enough that I could reason about the results when testing search behavior.My requirements were pretty straightforward:Use a real text dataset with broad coverageUse Python, partly for more hands-on practice with itStore the output in ParquetMake the output compatible with Pinecone bulk importSupport local embedding generationMake it possible to split the embedding work across multiple machines laterI landed on the stanford-oval/ccnews dataset from Hugging Face. It gives me a broad set of news articles, and it includes useful metadata fields like title, author, source URL, published date, and category information.For the Pinecone import format, the structure is simple:The field is the unique vector ID. I generally prefer structured IDs because they make debugging and operational tasks easier later. In this example, UUIDs are used, but this could easily be adjusted to include prefixes, document IDs, chunk numbers, or other identifiers.The values field contains the dense vector.The metadata field contains the JSON metadata associated with the vector. One thing to watch out for when writing Parquet for bulk import is that the metadata field should be written as a JSON string, not as a nested JSON object. That is a small detail, but it matters when you are preparing files for import.Choosing an Embedding ModelFor this example, I used BAAI/bge-large-en-v1.5, accessed through the registry name in .This model produces 1024-dimensional dense vectors. I could have used Pinecone-hosted embedding models, but for this project I wanted to experiment with local embedding generation. I also wanted to simulate a workflow I see frequently in the field, where customers generate embeddings themselves and then load the resulting vectors into Pinecone.There are tradeoffs here. Higher-dimensional vectors can improve retrieval quality in some workloads, but they also increase storage size and can affect speed and cost. Pinecone can support many different embedding models, so the right choice depends on what you are testing. If I were testing multimodal search, for example, I might use a model like CLIP instead.The model isn't a fixed choice here; what matters is that the workflow stays flexible enough to swap models later.First Version: Stream, Embed, and Upsert DirectlyThe first version of the script is the easiest one to understand. It streams the CC News dataset from Hugging Face, chunks the article text, generates embeddings locally, and upserts the vectors directly into a Pinecone index.This is useful for smaller tests because there are fewer moving parts. The script does everything in one flow: create the index if it does not exist, stream records, skip unusable or non-English records, chunk the text, generate embeddings, attach metadata, and upsert. The full script is in the notebook (section 1). A few details are worth calling out.First, the script checks the metadata size. Pinecone enforces a 40 KB limit on the metadata payload per vector, and since I am storing the original text in metadata for these tests, a long article can blow past it. The chunker trims iteratively until the serialized metadata fits:while sys.getsizeof(json.dumps({"text": chunk})) >= MAX_JSON_BYTES and len(chunk) > 1000: