Remove the files associated to the submodule

rm -rf .git/modules/path/to/sub-module-1

Remove any references to submodule in config and .gitmodules manually.

vim .git/config
vim .gitmodules

delete folder sub-module-1 folder and stage this change first

rm -rf path/to/sub-module-1
git add path/to/sub-module-1

then, you will add sub-module-1 again successfully.

git submodule add url path/to/sub-module-1