I am searching for a concept to distribute circles in a square randomly, so that they dont overlap. All circles are of the same size. The area covered by the circles can be high
Suppose you want n = 200 circles. My suggestion is to pick a number moderately larger than n, say m = 300, and generate that many points at random locations within the square. This set of m points is your candidate set of circle centres. Now produce a graph containing m vertices, one for each point, in which two vertices are linked by an edge if and only if the distance between their points is less than the circle diameter -- it is exactly in this situation that we would be forbidden to include both circles in the solution, because they would overlap.
You can now model the problem of choosing which candidate circle centres should actually become circles as a maximum independent set problem: find a maximum-sized set of vertices in this graph having the property that no two vertices in the set are linked by an edge. This will find a set of circle centres such that no two of their circles would overlap. If this set contains more than n circles, then discard a random selection of circles until just n remain. If fewer than n circles are found, you'll need to increase m and try again.
Unfortunately the maximum independent set problem is NP-hard, so I don't know whether it will be feasible to solve on a 300-vertex graph. (And in turn, I don't know whether 300 random centres will give you enough flexibility to find 200 non-overlapping circles...) But in any case, you would normally solve maximum independent set in one of two ways:
Those Wikipedia pages contain links to papers describing algorithms for these problems that, while still exponential-time, are much faster than a standard "full backtracking" algorithm. A couple of things to bear in mind: