should I include things that have no importance for the story?
Everything you include in your text is by definition part of the story and will affect the reading of every other part: Every part is automatically important whether you want it to be or not.
The factors to take into account when considering whether to include something are:
- is it interesting?
- is it relevant/consistent?
Consider two extreme cases:
An unrelated character who is about to die is described at length. The reader's mind may now be seeded with thoughts of death, mortality, futility, tranquility, etc. (depending on the nature of the description) while they read the next part of the text, and their interpretation of it will be coloured by these themes.
A book that consists of a slim plot, riddled with tangential and meandering asides. This has the effect of exposing the reader to a vast field of disparate subjects. They are free to draw their own connections between topics and become very active in their interpretation of the text. They will judge the action of the main plot within the context of the wider world. In fact, the plot can even be so slim as to appear non-existent: The best example I can give of this is the film Koyaanisqatsi, but The Instruction Manual by Julio Cortázar and Life A User's Manual by Georges Perec are good examples in literature. You may not even have to try to make the deviations relevant to each other: the human mind will find stories in the most minimal stimulus, as shown by the experiments of Lev Kuleshev.
These illustrate that relevance is a complex and sometimes unintuitive issue. Further, relevance trumps interestingness. Consider The Princess Bride by William Goldman, ostensibly a rewriting of another book in which "tedious" parts of the "original" text form a sub-plot about its author.