Your Toolbox is linked to your account, not a specific project. This allows you to access your Toolbox bits and materials across all your projects.
Deleting a custom or Inventables material from your Toolbox removes it from your Toolbox permanently (don't worry, you can always add it again). This will also remove the material from your other project files.
Note: once a custom material or bit is removed from your Toolbox, it is removed from your account and the material drop-down menu near your project settings. You can add custom or Inventables materials back to your Toolbox at any time, even if you have previously deleted them.