Discussion Forum

TypeDB insertion Error

Hi ,
I am using TypeDb 2.3.3 and I have created the schema using console and now I am trying to insert a farily large data.gql (99,721 kb to be exact) file through console. But its giving me error “Unable to connect to typeDb server”. I checked typerDb server is running.
The commands I am using are :

For creation of Schema :

database create KG
transaction KG schema write
source AllEntities_Scehma.gql
commit

For insertion of Data :

transaction KG data write
source KG.gql
commit

What could be the issue here ?

Console is not designed to ingest large datasets. Notably because it’s not going to be performant for scale - it’s designed for user’s manual interaction with the database, so it has pretty printing and other features that make it slow. Additionally all the queries in one source command happen within one transaction. What’s probably happening is that your server is running out of memory because the transaction becomes too large!

1 Like

So what should be the course of action ? I need to have multiple insert batches or do you recommned moving towards Python api for insertion ?
My plan is to insert using console somehow and will query afterwards with api. Is there a way i can do that big insertion with just Console ?

Your options are:

  • break your file into files with say 1k queries per file. Then script the console with sequence of:
transaction <db> data write
source <file-n>
commit
  • the more correct way to do this is to use a client to load the data, such as python or java or nodejs :slight_smile: