projects/gcnet/README.md
01Project · November 2016
02

GCNet

03
GIF Caption Network
05

GCNet reads a GIF and writes a sentence about what happens inside it. A dog catching a toy. A car rolling down a road. A soccer player burying a goal. It learned to describe motion, not just pictures.

06

It works the way you would write the sentence yourself: one word at a time. At each step the network looks at the whole GIF and the words it has written so far, then predicts the next word. Feed that word back in and repeat, and the caption finishes itself.

07

I built it in November 2016. That date is the most interesting thing about it.

08
KerasTheano / TensorFlowVGG16GloVePython
archived · November 2016github.com/chcaru/gcnet
notein-context.md

The world has changed since this ran

When GCNet was written, this was genuinely hard. The transformer paper was still seven months away. There was no CLIP, no GPT you could prompt, no foundation model that had ever been shown an image. If you wanted a machine to describe a moving picture, nobody handed you one. You assembled it yourself, layer by layer, and trained it from a pile of GIFs on a single graphics card in your room.

To place the moment: the first commit lands in late November 2016. That summer, Pokemon GO had emptied offices into the streets and the iPhone 7 had just killed the headphone jack. The most impressive artificial intelligence on the planet had beaten a human champion at Go that spring, and then it could do nothing else with itself.

That ceiling was narrow. That fall, Google had quietly moved Translate onto a neural network. DeepMind had taught a model to generate raw audio one sample at a time. The frontier of teaching a machine to describe a picture was a convolutional network handing off to a recurrent one, the lineage GCNet was born into. PyTorch did not exist yet; it would arrive a couple of months later, so the tools were Theano and a young TensorFlow, stitched together with Keras. There was no model you could simply ask. You chose a problem, gathered the few pieces that existed, and built the rest by hand.

GCNet walks that path in its own way. Take a convolutional network already fluent in photographs and use it as eyes. Take word vectors distilled from 840 billion tokens of text and use them as a vocabulary. Wire the two together with recurrent memory, expand every caption into hundreds of next-word guesses, and hope the gradients behaved. GCNet did all of that, and it captioned GIFs it had never seen well enough to make you smile.

Today a multimodal model does the same thing in a single call, with no training, and describes the GIF better than GCNet ever could. That is not a defeat. It is the whole point of what I write about now. The distance between wanting something from a machine and getting it keeps collapsing. GCNet is a fossil from the last stretch where you still had to build the bridge by hand, and building it by hand is how I came to understand the thing that would later fit inside a single line of a prompt.

  1. 2016GCNetyou are here
  2. 2017Transformers
  3. 2018BERT · GPT
  4. 2021CLIP
  5. 2023Multimodal LLMs
architecturegcnet.model
repeat × 15skip · GIF 2024GIF framesVGG16 features · 16 × 1000Bidirectional LSTM→ 16 × 2048LSTM→ 1024Σ frames→ 1000Encoded GIFconcat · 2024Caption so farword indices · 15GloVe embeddingfrozen · → 15 × 300Fuse words + GIFconcat · 15 × 2324Bidirectional LSTM→ 15 × 2048LSTMencoded caption · 1024Caption + GIFconcat · 3048Dense · softmax→ 6001 wordsnext word
inputrecurrenttransformconcatprediction

GCNet computes P(next word | GIF, caption so far). Two encoders feed one softmax. Run it once per word, feeding each guess back in, and a sentence writes itself.

walkthroughhow-it-reads-a-gif
  1. 01
    eyes

    See the motion

    Each of the sixteen frames becomes a thousand numbers from VGG16, a network already fluent in photographs. A bidirectional LSTM reads the frames forward and backward to feel the motion, while a parallel sum captures the gist all at once. Together they become the Encoded GIF.

  2. 02
    words

    Hold the sentence

    The words written so far turn into GloVe vectors, a vocabulary learned from 840 billion tokens of text. The Encoded GIF is then stapled onto every word position, so each step sees the picture and the half-written sentence at the same time.

  3. 03
    memory

    Compress to a thought

    A second bidirectional LSTM reads that fused sequence and squeezes it into a single vector: everything the network currently understands about this GIF and this caption-in-progress.

  4. 04
    guess

    Pick the next word

    That thought is joined with the Encoded GIF one final time, a shortcut so the picture is never forgotten, and a softmax over six thousand words places its bet on what comes next.

notableahead-of-its-time

What it got right before the field did

None of GCNet's parts were invented here. The convolutional eyes, the word vectors, the recurrent memory were all drawn from the field. What is worth noticing is the shape they formed. A few choices that felt like plain common sense in 2016 turned out to be the choices the whole field would make.

  1. 01

    Generation as a next-word loop

    GCNet never writes a full sentence at once. It answers a smaller question, given this GIF and the words so far, what comes next, then feeds its own answer back in and asks again. The caption assembles itself one word at a time.

    in contextThis is the autoregressive loop that now drives every large language model. In 2016 the common move was to unroll one recurrent decoder and carry its hidden state forward. GCNet re-read the whole context and placed a fresh bet at each step instead. The instinct the field would later standardize on, reached for before it had a name.

  2. 02

    One picture, injected everywhere

    The Encoded GIF is not shown to the language side once and then forgotten. It is fused onto every word position going in, and then handed to the final decision a second time through a shortcut that skips the language network entirely.

    in contextWhether to seed the image once or keep it present at every step was a live debate in 2016. That second shortcut to the output is a skip connection, the same trick that had just made very deep vision networks trainable the year before. Condition hard, re-inject often: ordinary in multimodal models now, a hunch back then.

  3. 03

    Two readings of one GIF

    Motion and content are different questions, so GCNet asks both. One path reads the sixteen frames in order, forward and backward, to feel how things move. A second path adds the frames together, ignoring order, to capture what is simply present. The two become a single description.

    in contextPairing an order-sensitive temporal read with an order-blind summary is the kind of small, pragmatic choice you only reach for by wrestling with real video. It never makes a headline. It is part of why the captions describe what happens, not just what is on screen.

  4. 04

    Freeze the experts, train the translator

    GCNet's eyes, a vision network, and its vocabulary, word vectors distilled from 840 billion tokens, were both frozen. Not one weight in either was touched. Only the piece in the middle, the part that learns to turn seeing into saying, was trained.

    in contextThis is now the default recipe for multimodal systems: take strong pretrained encoders, lock them, and train a small bridge between. In 2016 it was as much necessity as strategy, since there was no compute to do more. The constraint and the right answer turned out to be the same thing.

resultsgenerated-captions

Captions GCNet wrote for GIFs it had never seen during training. The words are the network's own.

A looping GIF that GCNet captioned: a monkey with an animal is eating something.
“a monkey with an animal is eating something”
A looping GIF that GCNet captioned: a soccer player is scoring a goal in a football match.
“a soccer player is scoring a goal in a football match”
A looping GIF that GCNet captioned: a white car is driving down a road.
“a white car is driving down a road”
A looping GIF that GCNet captioned: a man is holding a microphone and moving his hands.
“a man is holding a microphone and moving his hands”
A looping GIF that GCNet captioned: a dog is trying to catch a toy.
“a dog is trying to catch a toy”
A looping GIF that GCNet captioned: a girl with blonde hair is talking and moving her head.
“a girl with blonde hair is talking and moving her head”
A looping GIF that GCNet captioned: a young woman in a car is driving and smiles.
“a young woman in a car is driving and smiles”
try itrun-it-yourself

The trained network is about ninety million numbers. They are small enough, ten years on, to send to your browser and run on your own machine. Pick a GIF and GCNet will write a caption for it live. The model and your GIF never leave this tab.

To keep the download small, the weights here are quantized: squeezed to roughly a quarter of their original precision. A caption can come out a little different from the original, full-precision model.

Download the raw weights (v1.0.0)
A dog leaping to catch a toy.101123.gif
pick a gif

A one-time 217 MB download. It runs best on a desktop with a GPU. Everything happens locally.

buildthe-pipeline

Before a single word could be predicted, the data had to be built. End to end, on good hardware, this took most of a day.

  1. 01DownloadAround 100k GIFs from the TGIF dataset. Roughly 120 GB on disk.
  2. 02Split framesEvery GIF cut into sixteen PNG frames. Now roughly 250 GB.
  3. 03Clean captionsStrip punctuation and stray expressions into something learnable.
  4. 04Build vocabularyThe 6,000 most frequent words, matched to pretrained GloVe vectors.
  5. 05Precompute VGG161.65M frames pushed through VGG16. About three hours on a GTX 1080.
  6. 06TrainExpand each caption into next-word examples and learn the weights.
constraintsone-gpu-in-a-room

Everything had to fit

There was no cloud to rent by the hour and no cluster to borrow. GCNet was trained on a single GTX 1080, the best consumer graphics card you could buy in 2016, with eight gigabytes of memory to its name. That ceiling shaped every decision that followed.

The vision network was the first thing that did not fit. Running VGG16 across sixteen frames of every GIF, inside the training loop, next to the model actually being trained, would have overflowed those eight gigabytes and crawled even where it fit. So the eyes were peeled off and run once, ahead of time. All 1.65 million frames in the dataset were pushed through VGG16 over about three hours and saved to disk as compact fingerprints. Training read the fingerprints instead of the pixels, and the expensive part never ran twice.

The pressure did not stop at the GPU. The word vectors alone, learned from 840 billion tokens of text, took roughly forty of the machine's sixty-four gigabytes of RAM just to load. The raw GIFs were 120 gigabytes on disk; split into frames they passed 250. A good processor of the day spent hours doing nothing but decoding and resizing images. Every stage was a negotiation with a resource that was about to run out.

The architecture added its own tax. Because GCNet learns by predicting a single word at a time, every caption had to be expanded into one training example per word it held. A hundred thousand captions became more than 1.6 million rows. The elegant framing was paid for in memory and in patience, one epoch at a time.

  • GPUGTX 1080 · 8 GB VRAM
  • System memory64 GB RAM
  • Dataset on disk~250 GB of frames
  • Precompute~3 hrs · 1.65M frames

For everything it cost to train, the model itself was small. Roughly ninety million parameters, a number you can almost hold in your head. The same model today would load on a phone without noticing.

  1. ~90MGCNet 2016Fit on one consumer card. Trained in a room.
  2. 175BGPT-3 2020About 2,000 times larger.
  3. 1T+Frontier models todayEstimated in the trillions. A datacenter, not a desk.

GCNet did not get smaller. Everything around it became unimaginably larger.