How would I reverse engineer a cryptographic algorithm?

Question

I wrote an application that encrypts text in this way:

1. Get the input text

2. Reverse the text

3. Convert to hexadecimal

4. <

Answer 1

An attacker would probably (in general) do the followings:

Identify and defeat any 'visible for the eye encoding' or trivial crypto, like reversing of text, Base64 encoding, ROT13, ect.

At a point that they find a high entropy state they try to acquire more pieces of encoded data, and XOR them together. This results in an XOR-ing of the two original plain-text with the key cancelled out in case the encoding is indeed XOR based (like RC4) and the key was constant. If the attacker can get hold of any plain-text - encoded data combination, all other encoded data is decode-able.

At final desperation they may try to test against most common practices, like where they use RC4 or other simple algos with a dynamic key, and put the key on the end or beginning of the file/data.

If they only have access to encoded text, this is pretty much the end of the road. In case they have a access to.. like an API where they can produce the encoded version of a supplied plain-text, then they will trivially identify if its a bit based (like XOR), or block cypher, or feed forward block cypther encoding, but the getting the key and the actual algo is still a problem.

If they have access to the decoding program for symmetric key encoding (like your XOR), or the encoding program of asymmetric key encoding, the encoding is most likely instantly defeated by reversing it.