| Source text | Reconstruction (inverted from OpenAI embeddings) | Notes | | --- | --- | --- | | My research investigates the future of knowledge representation and creative work aided by machine understanding of language. I prototype software interfaces that help us become clearer thinkers and more prolific dreamers. | My research investigates the future of knowledge representation and creative work aided by machine understanding of language. I prototype software interfaces that help us become clearer thinkers and more prolific dreamers. | | | Mr and Mrs Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much. They were the last people you'd expect to be involved in anything strange or mysterious, because they just didn't hold with such nonsense. | The people who were Mr. and Mrs. Dudley were perfectly normal, just a matter of habit, and they were always sure that they didn't want to be visited. They were proud to say that, despite their strange devilish appearances, they were very familiar with each other and the real house, so hardly anything was wrong. | Got the general structure + topic right, including something close to “Dursley”. | | Taylor Swift is a globally acclaimed singer-songwriter known for her captivating storytelling through music. With a remarkable ability to connect with her audience, her versatility spans across genres and her lyrics resonate with fans of all ages, making her one of the most influential artists of her generation. | Taylor Swift is a singer and songwriter who is best known globally for her creative and influential storytelling. As one of the most widely recognized artists of all time, Swift is a globally renowned singer with a string of hit songs, capable of transforming the sway of audiences with her talent and wit, and she is increasingly known for her contributions to music across a diverse spectrum that always inspires new listeners. | Got her name (a specific proper noun) | | We humans are toolmakers by nature, but most of us can't build or modify the software we use every day — arguably our most powerful tool. Here at Notion, we're on a mission to make it possible for everyone to shape the tools that shape their lives. | Most of us are not a tool maker, but we are passionate about a way to make the most of our time. Ideally, to build and manipulate humankind's software, we want to create a software tool that is the fundamental unit of all life - not just the words on the walls, and our potential is now free to help us design something that works. | Caught the detail that there was a dash (”-”) in the text | | In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English. | In the Andes mountains, an international group of scientists, living in a remote valley in Peru, found a previously undiscovered flock of rare unicorns that speak a striking English language and are amazingly scaly. Despite promises that they could be found in the heart of the Andean mountains, in fact, the unpublished research revealed a wild female unicorn was living in the sediments, and has become an example of talking. | Got a few nontrivial proper nouns/keywords, like “Andes Mountains” and “English”, right | | Miley Cyrus was caught shoplifting from Abercrombie and Fitch on Hollywood Boulevard today. | Sheryl Crow was caught on MTV stealing from an Abercrombie and Fitch store in Hollywood, Pa. | Missed most of the proper nouns (maybe they’re too rare?) but got the general topic + tone. |
<aside> ⭐ Though these only show input and output with these features, I also validated that when these inputs without these features are reconstructed, those features aren’t present in the reconstructed output.
</aside>
| Source text | Reconstructions (inverted from OpenAI embeddings, emphasis mine) | Feature reconstructed / Notes | | --- | --- | --- | | today is the first day of the rest of your life. | today is the first day of a new life. Today, you will consider the rest of your life. | Lowercasing | | The United States and Russia were locked in a silent but heated competition — the Cold War. | The Cold War was a fierce but secretive competition between the United States and Russia—the Cold War was often fought for Black Lightning. | Em-dash/hyphens (I’ve found that embeddings generally represent the presence of all kinds of punctuation (parens, question marks, etc.) quite well. | | My name is Jackie McMillan, and my favorite color is purple. | My favorite name is Jackie McQuillan, and I love blue and red. | Names | | The author was born and raised in West Lafayette, Indiana and attended University of California, Berkeley. | He was born and raised in West Lafayette, Indiana, and graduated from the University of Chicago and the University of California, Berkeley. | Specific locations and names | | In August 2012, Voyager 1 made the historic entry into interstellar space, the region between stars, filled with material ejected by the death of nearby stars millions of years ago. | In August 2012, before the star Vanguard 1 entered the astronomical universe, the International Space Station made history by sending out millions of light years of cosmic dust, ejected from the spacecraft between its entrance and the star, into the region that had become an asteroid on July 1, 2013, and returning to Earth with the lost vitality of many ancient objects.
In August 2013, near the end of its orbit, the Space Shuttle Columbia made history when the voluminous spacecraft, launched by the United States on August 1, 2012, passed through the region of interstellar space containing many billions of years of old material, making a return to the surface, and into the ancient star system that had attracted the attention of visitors and the universe. | Specific dates and times |
<aside> ⭐ My current adapter and decoder model are trained on a pretty limited domain (English Wikipedia), so it’s quite easy to push the inputs out of distribution.
</aside>
| Source text | Reconstruction (inverted from OpenAI embeddings) | Notes | | --- | --- | --- | | Scaling Laws for Neural Language Models | These Scaling Limits are based on First Languages and Large Scales. As a simple example, I'm trying to scale the data to a Neural Expression. The following rules are modeled for Moving Languages to be 1.0 / g (grammar) for Large Scales. | On some inputs the model outputs something super long/obviously incorrect, though the topic usually stays similar. | | My name is Linus Lee. | I My name is Lee Lin.
I My name is Lee Lino. | Sometimes, reconstruction fails even for simple, short inputs. | | OpenAI today announced GPT-4, an LLM with high performance on benchmarks such as GSM8K and BigBench as well as standardizes tests like SAT, GMT, LSAT, and the ACT. | Google LP has announced GAT4M, a big-text search engine that can perform high quality tests, including ALGOL and PSAT, and it is today launching the OpenSecrets 4 test with options for MPI, byte-code and popular LAST algorithms. | Struggles with heavily technical or jargon-y inputs. |
Input
In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.
Outputs, sampled at temperature = 0.75
Of course, in the Andes Mountains, an international research group discovered a strange unicorn that lived in an uncharted valley in Peru, and, despite reporting no signs of speaking English, they were able to find it. In a surprise revelation, scientists from a leading British university have found a living and perfectly cultivated race of ancient mountain lions, described as wild in the English language, which are still appearing in the Andean deserts. The unicorn spoke an unusual language.
In the Andes mountains, an extraordinary group of researchers found a rare, undiscovered unicorn that has been living in a valley in Peru, and despite protests from the local linguists, they were able to speak English. More recently, a scientific investigation revealed that the unicorns, found in a mountainous region of the American West, appear in a sparkling white voice and are already speaking natural languages, frightening the Peruvian authorities.
In the Andes, an international research group found that a rare and spectacular genus of unicorns, living in the highlands of Peru, had just escaped the English language and are still speaking it. According to unconfirmed reports, the strange unicorns appeared in a living river valley, and scientists were able to discover a fresh egg that features most of the human race. There is no doubt in Spanish.
Of course, in the Andean mountains, an international research team discovered a rare cryptic white squirrel that live in the mountains of Peru, and despite appearing unfamiliar in English, they were found speaking a variety of languages. In a stunning revelation, scientists and nuns have visited a lost village in the Sierra Nevada foothills, finding evidence that the rare creature simply spoke a living language.
In the Andes mountains, an unofficial research group of Spanish scientists discovered that a rare crowned unicorn, speaking a strange language, was living in abundance in the wild in the valleys of the Andes. Despite initial reports that they were found in an English village, scientists and people have now been able to find a stunningly expressive unicorn that speaks natural languages, leaving a mystery in the process.