Really slow drop of indexes

[johny65]

Is it normal to take so much time to drop an index? I have an index over 3 millions of vertices, dropping it takes almost 7 minutes:

Server side script executed in 270,993988 sec(s). Value returned is: [][Started multi-line command. Type just 'end' to finish and execute]

I have a migrated database with several auxiliary indexes for the migration process, deleting all of those auxiliary indexes takes an hour or more. Even so, if I don't run the drop commands in separate scripts, then all my RAM is exhausted. Is this an issue or is it normal? (Using 3.1.x version).

[luigidellaquila]

Hi @johny65

I'd say it's not normal, index drop should take just a few seconds.

Any chance to have a thread dump of OrientDB server while the index drop is happening?

Thanks

Luigi

[johny65]

Hi @luigidellaquila. Here are the thread dumps of both the console and the server while a index drop is happening. It took approximately 12 minutes:

orientdb> connect remote:localhost/mydb admin admin;
IMPORTANT! Using default password is unsafe, please change password for user 'admin' on database 'mydb'
Connecting to database [remote:localhost/mydb] with user 'admin'...OK
orientdb {db=mydb}> drop index the_index;
Removing index...
Dropped index in 728,041992 sec(s).
Index removed successfully

console_threaddump-1620299243328.tdump

server_threaddump-1620299223362.tdump

[luigidellaquila]

Hi @johny65

Thank you very much, I checked the thread dumps and yes, it's iterating the index to remove it.
Looking at the history of that component, it was implemented like that because of durability issues.
I'll discuss it with the storage team ASAP to see how we can optimise it

Thanks

Luigi

[johny65]

Ok! Thank you @luigidellaquila.

[andrii0lomakin]

Hi guys,
That is true if we delete the index we can not merely delete a file, which will not allow us to restore database consistency after the crash.
We need to delete each record from the index and then delete an empty file so if for example database is crashed into the middle of the index drop it will be restored properly.

[andrii0lomakin]

In future, we probably will merely log all key entires into the index and will delete the index file itself which is obviously faster than delete keys one by one. But now that is not a priority for us.

[timw]

@Laa - having a look at 3.1.x, the index file deletes are already logged in the atomic operation (and appear to be rolled-forward in the recover code), so are the individual index entry deletes redundant?

[andrii0lomakin]

Hi @timw , I suppose so. But we are now busy with other tasks and will investigate possibilities to speed up the deletion of the index later.

[sergey-sotnikov]

Good afternoon
I have a db with 422 million vertices.
and deleting the index takes forever.
can anything break if i delete the folder with this lucene index instead of executing drop command?
or is there any way to remove this index faster?

thanks!

PS version 2.2.22 (i connect to this db via plocal in console.sh)

[timw]

I can't speak to 2.2, but in 3.1 you can simply delete the Lucene index folder (and then perform a rebuild index).

[johny65]

I started this discussion (and associated issue #9637) referring to no-Lucene indexes. I have no problem deleting Lucene indexes, it is fast (but not rebuilding them).

[sergey-sotnikov]

I can't speak to 2.2, but in 3.1 you can simply delete the Lucene index folder (and then perform a rebuild index).

ok, thanks
it works