Discussion Forum

[How to] save database and move it around

Once insert and commit some transations of grakn, I’d like to save it on disk and move the graph database to other machines. I couldn’t find any information in Doc and Discussion forunm. Any one have any ideas?

Appreciate the help!

Thanks to 1.8.1, based on release note, the ability to export and import date and schema is added.

grakn server export [--no-anim] <keyspace> <file>.grakn
grakn server import [--no-anim] <keyspace> <file>.grakn

So I tried it, unlucky, there’re some errors with import.
export looks fine:

sz@pc:~$ ./grakn1.8.1/grakn server export baike1 baike1.grakn
$x isa export,
    has status "completed",
    has progress (100.0%),
    has count (3,916 / 3,916);

import meets error:

sz@pc:~$ ./grakn1.8.1/grakn server import baike12 baike1.grakn
$x isa import,
    has status "in progress",
    has progress (0.0%),
    has count (0 / 3,916); -
An error has occurred during boot-up. Please run 'grakn server status' or check the logs located under the 'logs' directory.
io.grpc.StatusRuntimeException: INTERNAL: java.lang.NullPointerException

logs:

2020-08-26 16:58:55,777 [migration-executor-thread] INFO  grakn.core.server.migrate.Export - Exporting baike1 from Grakn 1.8.1
2020-08-26 16:59:11,506 [migration-executor-thread] INFO  grakn.core.server.migrate.Export - Exported 1761 entities, 1204 attributes, 951 relations (4317 roles), 5428 ownerships
2020-08-26 16:59:59,360 [migration-executor-thread] INFO  grakn.core.server.migrate.Import - Importing baike1 from Grakn 1.8.1 to baike12 in Grakn 1.8.1
2020-08-26 16:59:59,369 [grpc-request-handler-4] ERROR grakn.core.server.rpc.MigrateService - An error occurred during import.
java.util.concurrent.ExecutionException: java.lang.NullPointerException
        at grakn.core.server.migrate.AbstractJob.awaitCompletion(AbstractJob.java:85)
        at grakn.core.server.rpc.MigrateService.runJob(MigrateService.java:126)
        at grakn.core.server.rpc.MigrateService.importFile(MigrateService.java:102)
        at grakn.core.server.migrate.proto.MigrateServiceGrpc$MethodHandlers.invoke(MigrateServiceGrpc.java:398)
        at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:172)
        at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)
        at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:814)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: null
        at grakn.core.server.migrate.Import.insertEntity(Import.java:273)
        at grakn.core.server.migrate.Import.executeInternal(Import.java:201)
        at grakn.core.server.migrate.AbstractJob.execute(AbstractJob.java:65)
        ... 3 common frames omitted
2020-08-26 16:59:59,378 [main] ERROR grakn.core.daemon.GraknDaemon - An error has occurred during boot-up. Please run 'grakn server status' or check the logs located under the 'logs' directory.
java.lang.RuntimeException: io.grpc.StatusRuntimeException: INTERNAL: java.lang.NullPointerException
        at grakn.core.daemon.GraknDaemon.import_(GraknDaemon.java:326)
        at grakn.core.daemon.GraknDaemon.run(GraknDaemon.java:186)
        at grakn.core.daemon.GraknDaemon.main(GraknDaemon.java:85)
Caused by: io.grpc.StatusRuntimeException: INTERNAL: java.lang.NullPointerException
        at io.grpc.Status.asRuntimeException(Status.java:533)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:442)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:510)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:630)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:518)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:692)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:681)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Hi yiouyou,

This error occurs because a type that was exported does not exist in the schema of the database you are importing to. In this case, it is an Entity Type that is missing. The error reporting of migration importer is not very useful right now, but you can assume that any NullPointer indicates that a label in the exported data set does not exist in the schema of the new database.

In this case, please check that the schema you imported matches the schema in the original database exactly. If you do need to rename a label, there is a feature of import described here: https://dev.grakn.ai/docs/management/migration-and-backup#incompatible-schema

@adam Thanks for your tips and help. I thought the *.grakn will contain the schem it self, and it was wrong. Before importing *.grakn, setting schema is still a necessary step.

I fixed the issue and the importing is in process but some other errors come up and the process couldn’t be completed.

sz@air:~$ ./grakn1.8.1/grakn console -k baike1 -f baike1_schema.gql
Loading: /mnt/d/coding/guanguan_kg/baike1_schema.gql
...
{}
Successful commit: baike1_schema.gql
sz@air:~$ ./grakn1.8.1/grakn server import baike1 baike1.grakn
$x isa import,
    has status "in progress",
    has progress (76.9%),
    has count (3,012 / 3,916); -
An error has occurred during boot-up. Please run 'grakn server status' or check the logs located under the 'logs' directory.
io.grpc.StatusRuntimeException: INTERNAL: java.lang.ClassCastException: grakn.core.concept.impl.EntityImpl cannot be cast to grakn.core.kb.concept.api.Attribute

logs:

2020-08-26 21:24:34,559 [main] INFO  grakn.core.server.Grakn - Grakn started in 1.259 s
2020-08-26 21:25:09,819 [migration-executor-thread] INFO  grakn.core.server.migrate.Import - Importing baike1 from Grakn 1.8.1 to baike1 in Grakn 1.8.1
2020-08-26 21:25:17,767 [grpc-request-handler-13] ERROR grakn.core.server.rpc.MigrateService - An error occurred during import.
java.util.concurrent.ExecutionException: java.lang.ClassCastException: grakn.core.concept.impl.EntityImpl cannot be cast to grakn.core.kb.concept.api.Attribute
	at grakn.core.server.migrate.AbstractJob.awaitCompletion(AbstractJob.java:85)
	at grakn.core.server.rpc.MigrateService.runJob(MigrateService.java:126)
	at grakn.core.server.rpc.MigrateService.importFile(MigrateService.java:102)
	at grakn.core.server.migrate.proto.MigrateServiceGrpc$MethodHandlers.invoke(MigrateServiceGrpc.java:398)
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:172)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:814)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassCastException: grakn.core.concept.impl.EntityImpl cannot be cast to grakn.core.kb.concept.api.Attribute
	at grakn.core.server.migrate.Import.insertMissingAttributeOwnershipsAtEnd(Import.java:368)
	at grakn.core.server.migrate.Import.executeInternal(Import.java:213)
	at grakn.core.server.migrate.AbstractJob.execute(AbstractJob.java:65)
	... 3 common frames omitted
2020-08-26 21:25:17,791 [main] ERROR grakn.core.daemon.GraknDaemon - An error has occurred during boot-up. Please run 'grakn server status' or check the logs located under the 'logs' directory.
java.lang.RuntimeException: io.grpc.StatusRuntimeException: INTERNAL: java.lang.ClassCastException: grakn.core.concept.impl.EntityImpl cannot be cast to grakn.core.kb.concept.api.Attribute
	at grakn.core.daemon.GraknDaemon.import_(GraknDaemon.java:326)
	at grakn.core.daemon.GraknDaemon.run(GraknDaemon.java:186)
	at grakn.core.daemon.GraknDaemon.main(GraknDaemon.java:85)
Caused by: io.grpc.StatusRuntimeException: INTERNAL: java.lang.ClassCastException: grakn.core.concept.impl.EntityImpl cannot be cast to grakn.core.kb.concept.api.Attribute
	at io.grpc.Status.asRuntimeException(Status.java:533)
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:442)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:510)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:630)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:518)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:692)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:681)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

by the way, I double checked the schema and my insertion script, and I couldn’t find any silly error. “grakn.core.concept.impl.EntityImpl cannot be cast to grakn.core.kb.concept.api.Attribute”, what does this mean?

Any help will be appreciated.

More information for someone to debug:

The encoding of *.grakn on my system is Western (Windows 1252). If I change the encoding to UTF-8, the error comes up.

io.grpc.StatusRuntimeException: INTERNAL: com.google.protobuf.InvalidProtocolBufferException: Protocol message had invalid UTF-8.

It seems “UTF-8” is not a valid encoding for export file of grakn.

Hi Yiouyou,

The .grakn file is in a binary delimited protobuf format, not text, so changing the encoding would modify the internal messages and make them unreadable (which is why Protobuf throws the exception.)

As for the grakn.core.concept.impl.EntityImpl cannot be cast to grakn.core.kb.concept.api.Attribute issue, I’ve had a look into this and it’s almost certainly a bug on our end that wasn’t brought up during our testing, we are preparing a fix.

The related schema, insertion query commands and exported *.grakn file are uploaded to the following address, in case you might need them to debug.

File link: https://pan.baidu.com/s/1MaO2y-u9-_Z1uAcnCCnjtQ
Extraction code: hsl9

Hi @yiouyou we have released a new version of Grakn v1.8.2 that should have fixed this casting issue! Mind giving it another shot?

Cheers!
Joshua

@joshua No luck yet, I haven’t try 1.8.2, but with the latest 1.8.3, it shows error as below:

cmd:

(py36-grakn181-rasa11012) sz@pc:~$ ./grakn1.8.3/grakn server version
1.8.3
(py36-grakn181-rasa11012) sz@pc:~$ ./grakn1.8.3/grakn server status
Storage: RUNNING
Grakn Core Server: RUNNING
(py36-grakn181-rasa11012) sz@pc:~$ ./grakn1.8.3/grakn server import baike1_ baike1.grakn
$x isa import,
    has status "starting"; \
An error has occurred during boot-up. Please run 'grakn server status' or check the logs located under the 'logs' directory.
io.grpc.StatusRuntimeException: INTERNAL: java.lang.NullPointerException

error:

2020-09-24 12:07:31,067 [migration-executor-thread] INFO  grakn.core.server.migrate.Import - Importing baike1 from Grakn 1.8.1 to baike1_ in Grakn 1.8.3
2020-09-24 12:07:31,076 [grpc-request-handler-4] ERROR grakn.core.server.rpc.MigrateService - An error occurred during import.
java.util.concurrent.ExecutionException: java.lang.NullPointerException
	at grakn.core.server.migrate.AbstractJob.awaitCompletion(AbstractJob.java:85)
	at grakn.core.server.rpc.MigrateService.runJob(MigrateService.java:126)
	at grakn.core.server.rpc.MigrateService.importFile(MigrateService.java:102)
	at grakn.core.server.migrate.proto.MigrateServiceGrpc$MethodHandlers.invoke(MigrateServiceGrpc.java:398)
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:172)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:814)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: null
	at grakn.core.server.migrate.Import.insertEntity(Import.java:273)
	at grakn.core.server.migrate.Import.executeInternal(Import.java:201)
	at grakn.core.server.migrate.AbstractJob.execute(AbstractJob.java:65)
	... 3 common frames omitted
2020-09-24 12:07:31,086 [main] ERROR grakn.core.daemon.GraknDaemon - An error has occurred during boot-up. Please run 'grakn server status' or check the logs located under the 'logs' directory.
java.lang.RuntimeException: io.grpc.StatusRuntimeException: INTERNAL: java.lang.NullPointerException
	at grakn.core.daemon.GraknDaemon.import_(GraknDaemon.java:326)
	at grakn.core.daemon.GraknDaemon.run(GraknDaemon.java:186)
	at grakn.core.daemon.GraknDaemon.main(GraknDaemon.java:85)
Caused by: io.grpc.StatusRuntimeException: INTERNAL: java.lang.NullPointerException
	at io.grpc.Status.asRuntimeException(Status.java:533)
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:442)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:510)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:630)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:518)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:692)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:681)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Thank you for uploading the reproducible example, we’ll look into this next week.

Hi @yiouyou, we’re looking into your issue. To help with the troubleshooting, can you provide the schema file? baike1.grakn only contains the data.

And this is actually one good thing to check. The migrator only migrate the data not schema, therefore you need to load the schema into the keyspace first before trying to import.

Wow, it’s working after loading the schema. @rui thanks very much!

sz@pc:~$ ./grakn1.8.3/grakn console -k baike1 -f /mnt/d/guanguan_kg/baike1_schema.gql
Loading: /mnt/d/guanguan_kg/baike1_schema.gql
...
{}
Successful commit: baike1_schema.gql
sz@pc:~$ ./grakn1.8.3/grakn server import baike1 baike1.grakn
$x isa import,
    has status "completed",
    has progress (100.0%),
    has count (3,957 / 3,957);

If the error msg is more specific, that would be very helpful, considering my problem was so stupid~

1 Like

@yiouyou Great! The error message is definitely something we can improve in the future!