The word translated rejoice in these verses is kauchaomai. It means "boast, glory, pride in oneself, brag." It is the same word he used in Rom. 2:17 and 2:23 when he spoke to Jews who boasted in their possession of the Law of Moses. There is no ground for boasting on the basis of the Law, since as Paul shows in Romans, the Jews failed to keep the Law and could only find condemnation on the basis of the Law.
But through the death of Jesus we can have peace with God (Rom. 5:1), and it is on the basis of the work of Christ that we can rejoice/boast. We don't have anything to brag about in ourselves - but we do have a great Savior, and He is worthy of bragging on! Later in Romans Paul strings together a series of boasts about God:
What then shall we say to these things? If God is for us, who can be against us? He who did not spare his own Son but gave him up for us all, how will he not also with him graciously give us all things? Who shall bring any charge against God’s elect? It is God who justifies. (Rom. 8:31-33)Even when Paul says in 5:3 that we rejoice/boast in our sufferings, it is not because of any human achievement, but rather because suffering leads to hope, "and hope does not put us to shame, because God's love has been poured into our hearts through the Holy Spirit who has been given to us" (5:5). We rejoice/boast "in hope of the glory of God" (5:2).
The late pitcher Dizzy Dean once said, "It ain't bragging if you can do it!" Christians can brag on
God because He has done it, and He will do it.