{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Training PyTorch models on an IPU using Paperspace Gradient\n", "\n", "These notebooks are designed to help you solve the problems you face, by leveraging the power of deep learning and the speed of the Graphcore IPU. These examples use PopTorch, a set of extensions to PyTorch to enable PyTorch models to run on Graphcore's IPUs.\n" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "
How to use PyTorch on the IPU\n", "\n", "The examples in the `learning-PyTorch-on-IPU` folder provide an introduction to using PopTorch to run PyTorch models on the IPU.\n", " \n", "\n", "
" ] }, { "attachments": {}, "cell_type": "markdown", "id": "e0c01f52", "metadata": {}, "source": [ "
Explore our advanced fine-tuning notebooks \n", "\n", "Fine-tune the GPT-J 6B model and explore how a smaller LLM can solve language understanding tasks efficiently, by running this notebook:\n", " \n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "
\n", " gptj-text-generation/
\n", "      finetuning.ipynb This notebook demonstrates how to fine-tune a GPT-J model downloaded from the Hugging Face Hub on the MNLI dataset.
\n", "      GPTJ-generative-inference.ipynb Use structured and few-shot prompting to tackle language understanding tasks with text generation powered by GPT-J.
\n", "\n", "



\n", "\n", "Fine-tune the BERT-Large model in our Natural-Language-Processing notebook, by running this notebook:\n", " \n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "
\n", " finetuning-bert/
\n", "      Fine-tuning-BERT.ipynb This notebook demonstrates how to fine-tune a pre-trained BERT-Large model on the SQuADv1 question answering task with PyTorch on the IPU.
\n", "\n", "



\n", "\n", "Fine-tune chest x-rays in our Vision Transformer (Vit) notebook:\n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "
\n", " vit-model-training/
\n", "      walkthrough.ipynb This tutorial will walk you through the process of fine-tuning a Hugging face vision transformer (ViT) model to detect multiple diseases from chest X-rays.
\n", "\n", "
" ] }, { "attachments": {}, "cell_type": "markdown", "id": "22390069", "metadata": {}, "source": [ "
Use the IPU to train GNN's\n", "\n", "Explore training GNN's for different tasks within the following notebooks:\n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "
\n", " distributed-kge/
\n", "      KgeModelling.ipynb The Knowledge Graph Embedding model notebook will show you how to train a KGE model on the IPU using the ogbl-wikikg2 dataset.
\n", " schnet-graph-network/
\n", "      PyG-SchNetGNN.ipynb The Prediction of Molecular Properties using SchNet notebook demonstrates training a SchNet graph neural network with PyTorch Geometric (PyG) on the Graphcore IPU.
\n", " temporal-graph-networks/
\n", "      Train_TGN.ipynb The Training Dynamic Graphs with Temporal Graph Networks notebook demonstrates how you can train a temporal graph networks (TGNs) on the IPU to predict connections in a dynamically evolving graph.
\n", "\n", "







\n", " \n", "\n", "Check out our award winning results on the OGB link prediction task for the [2022 Open Graph Benchmark Large Scale Challenge](https://ogb.stanford.edu/neurips2022/).\n", "Read our blog, [Accelerating and Scaling Temporal Graph Networks on the Graphcore IPU](https://www.graphcore.ai/posts/accelerating-and-scaling-temporal-graph-networks-on-the-graphcore-ipu) to learn more about our TGN model.\n", "\n", "
" ] }, { "attachments": {}, "cell_type": "markdown", "id": "d53dc50f", "metadata": {}, "source": [ "
Generate text and perform language understanding tasks on the IPU\n", "\n", "Fine-tune the GPT-J 6B and BERT-Large models to explore how a smaller LLMs can solve language understanding tasks efficiently, by running these notebooks:\n", " \n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "
\n", " gptj-text-generation/
\n", "      finetuning.ipynb This notebook demonstrates how to fine-tune a GPT-J model downloaded from the Hugging Face Hub on the MNLI dataset.
\n", "      GPTJ-generative-inference.ipynb Use structured and few-shot prompting to tackle language understanding tasks with text generation powered by GPT-J.
\n", " finetuning-bert/
\n", "      Fine-tuning-BERT.ipynb This notebook demonstrates how to fine-tune a pre-trained BERT-Large model on the SQuADv1 question answering task with PyTorch on the IPU.
\n", "\n", "



\n", "\n", "You can find additional NLP examples of IPUs that will run in Gradient at https://github.com/gradient-ai/Graphcore-HuggingFace\n", "\n", "" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Useful tips\n", "Find some useful tips within our tutorial on Managing IPU resources.\n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "\n", " \n", "\n", "\n", "
\n", " useful-tips/
\n", "     managing_ipu_resources.ipynb See how to monitor IPU usage, how to release IPUs when you are not using them, and how to re-attach your model to the IPU to use it again.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "\n", "#### Join our slack community and interact with our developers!\n", "\n", "\n", "[![Join our Slack Community](https://img.shields.io/badge/Slack-Join%20Graphcore's%20Community-blue?style=flat-square&logo=slack)](https://www.graphcore.ai/join-community)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" }, "vscode": { "interpreter": { "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1" } } }, "nbformat": 4, "nbformat_minor": 5 }