Bentley-Ottmann algorithm works for finding intersections of set of straight lines. But I have lot of polylines:
Adding to Geom's suggestion (an R-Tree is the way to go), further performance improvements can be gained by doing the following:
1. Simplify the polyline - The number of points in the polyline can be reduced while keeping the polyline's general shape. This can be done using an angle threshold and processing each point or by using the Ramer-Douglas-Peucker algorithm. Depending on what you're doing, you may need to keep track of which points from the original polyline were used as the start/end points for each segment of the simplified polyline (the indices of the original polyline's points will need to be stored somewhere).
In this example you can see how a polyline's number of points can be reduced. The red points indicate points which were not used from the original polyline, and the green points indicate points which were kept to build the simplified polyline.
2. Store the simplified polylines in an R-Tree, and determine the intersections between each segment of each polyline (comparing bounds of segments to reduce calculations is beneficial to performance). As this is being done, the old indices from the original polyline's segments are stored as information relating to each detected intersection, along with which polylines intersected (some sort of identifier can be used). This essentially gives you the start and end bounds of each segment in the original polylines where the intersections exist with each other polyline.
3. This step is performed only if the intersection location must match the exact location of the intersections of the original polylines. You will need to go back and use the original non-simplified polylines, along with the data from the intersection information obtained in step 2. Each intersection should have a start and end index associated with it, and these indices can be used to determine which specific segments of the original polyline need to be processed. This will allow you to only process the necessary segments (given by the start and end indices stored with the intersection information). An alternative to that would be to use the point itself and extend a bounding box outward, then process segments of the orignal polylines which intersect with that bounding box (although this will likely take longer).
4. It may be necessary to take an additional step to check the endpoints of each polyline against every other polyline's segments, since the simplification process can knock out some endpoint intersections. (this is generally pretty fast).
This algorithm can be further improved by using the Bentley-Ottmann algorithm (this is the sweep-line algorithm Geom was referring to). Also note that depending on the simplification algorithm used and the parameters used for such algorithms (angular tolerance for example), there may be a trade-off between performance and accuracy (some intersection results can be lost depending on how simple the polylines are).
Obviously, there are libraries out there which may be viable, but if you're limited by license terms due to the company you work for or the product you're working on, third party libraries may not be an option. Additionally, other libraries may not perform as well as may be required.
The sweep line algorithm has a nice theory but is hard to implement robustly. You need to treat vertical segments, and there might be cases where more than two line segments intersect in a single point (or even share a common line segment).
I'd use an R-Tree to store bounding boxes of the line segments of the polyline and then use the R-Tree to find possibly intersecting elements. Only these need to be tested for intersection. The advantage is that you can use a separate R-Tree for each polyline and thus avoid detection of selfintersections, if needed.
Consider using CGAL's exact predicates kernel to get reliable results.